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Abstract 

Media literacy interventions offer promising avenues for the prevention of risky health behaviors among 
children and adolescents, but current literature remains largely equivocal about their efficacy. The 
primary objective of this study was to develop and test theoretically-grounded measures of audiences’ 
degree of engagement with the content of media literacy programs based on the recognition that 
engagement (and not participation per se) can better explain and predict individual variations in the 
effects of these programs. We tested the validity and reliability of a measure of engagement with two 
different samples of 10 th grade high school students who participated in a pilot and actual test of a brief 
media literacy curriculum. Four message evaluation factors (involvement, perceived novelty, critical 
thinking, personal reflection) emerged and demonstrate acceptable reliability. 
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Media literacy interventions offer relatively new and promising avenues for the prevention of 
risky health behaviors among children and adolescents, but current literature remains largely 
equivocal about their efficacy (Banerjee & Kubey, 2013; Bergsma & Carney, 2008; Jeong, Cho, & 
Hwang, 2012; Martens, 2010). We propose that (a) much of this ambiguity stems from the lack of 
conceptual clarity in the literature regarding the cognitive process through which media literacy 
interventions influence their target audience, and (b) that the ability to track this cognitive process by 
means of valid and reliable measures is necessary to evaluating the effects of media literacy programs 
on their audience. Accordingly, the primary objective of this study was to develop and test 
theoretically-grounded measures of audiences’ degree of engagement with the content of media 
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literacy programs based on the recognition that engagement (and not participation per se) can better 
explain and predict individual variations in the effects of these programs. 

Media Literacy as Health Prevention Tool 

Media literacy is defined as the ability to access, analyze, evaluate and create messages in a wide 
variety of media modes and fonnats while recognizing the role and influence of media in society 
(Aufderheide & Firestone, 1993; Hobbs, 1998). Media literacy-based interventions have more recently 
been introduced into the field of health education and promotion and are currently recommended by 
leading health prevention organizations (e.g., the American Academy of Pediatrics, the Centers for 
Disease Control and Prevention, and the Office of the National Drug Control Policy) as means of 
educating adolescents about a range of negative media influences on their behavior (Primack, Fine, 
Yang, Wickett, & Zickmund, 2009). Such interventions typically include discussion of persuasive 
media strategies and analysis of sample persuasive media messages, sometimes along with a component 
that has students plan and/or produce counter-messages (see active involvement interventions, Greene, 
2013). These activities are intended to help adolescents become more aware of the constructed nature of 
messages and teach them to be more adept at identifying the motives, purposes, and points of view 
rooted in messages (Hobbs, 1998). 

Three recent systematic reviews (Banerjee & Kubey, 2013; Bergsma & Carney, 2008; Martens, 
2010) and one meta-analytic review (Jeong, Cho, & Hwang, 2012) sought to assess the efficacy of 
media literacy interventions based on existing evidence. The meta-analytic review investigated the 
average effect size and moderators of 51 media literacy interventions on topics such as advertising, 
alcohol, body image and eating, sex, tobacco, and violence. Overall, the mean effect size of media 
literacy interventions, weighted by sample size, was .37 (p < .001), with a 95% confidence interval 
ranging from .27 to .47. This finding demonstrates that media literacy interventions can have positive 
effects on knowledge, critical thinking, and evaluations of behaviors, and therefore, may be an effective 
approach for reducing potentially harmful effects of media messages (Jeong et al., 2012). For example, 
Banerjee and Greene’s (2006, 2007) tobacco prevention media literacy intervention was effective in 
reducing positive attitudes toward smoking and behavioral intention to smoke. Similarly, Austin and 
colleagues’ (2005) media literacy intervention to prevent tobacco use was successful in reducing 
youths’ beliefs that most peers use tobacco, increased their understanding of advertising techniques, and 
increased their levels of efficacy regarding the extent to which they would participate in advocacy and 
prevention activities (Austin, Pinkleton, Hust, & Cohen, 2005). Overall, then, there is good evidence 
that media literacy interventions can enhance the critical thinking abilities of young adults and 
encourage them to make healthier decisions. 

Unfortunately, not all media literacy programs are efficacious. Some programs report limited 
effects of media literacy interventions, while others report unintended effects or effects in an 
undesired direction (Banerjee & Kubey, 2013; Bergsma & Carney, 2008; Martens, 2010). For 
instance, in their study, Ramasubramanian and Oliver (2007) explored the role of media literacy 
training and counter-stereotypical media exemplars in decreasing prejudicial responses toward Asian 
Indians, African Americans, and Caucasian Americans. Participants were exposed to a media literacy 
video or a control group video, followed by an activity to analyze stereotypical or counter¬ 
stereotypical news stories. Media literacy video increased prejudicial responses as compared to the 
control video, thereby activating prejudice instead of decreasing such feelings (Ramasubramanian & 
Oliver, 2007). Other scholars have also demonstrated that improving children’s understanding of 
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television may heighten the salience or appeal of violence and increase children’s willingness to use 
violence (Byme, Linz & Potter, 2009; Nathanson, 2004). 

One possible explanation for the differential success of media literacy interventions involves 
variations in content and/or delivery across programs. The Center for Media Literacy (2007) endorses 
five core concepts as central to the design of the content of media literacy interventions: (1) all media 
messages are social constructions (i.e., constructed by somebody and never able to reflect reality 
entirely); (2) people who make media messages use creative languages that have rules (i.e., creative 
components such as words, music, movement, camera angle and others are utilized to develop a media 
message in different fonnats such as magazine cover, advertisement, etc.); (3) different people 
experience the same media messages differently; (4) producers of media messages have their own 
values and points of view; and (5) media messages are constructed to achieve a purpose, usually for 
profit and/or power. In a systematic review of health-promoting media literacy interventions, Bergsma 
and Carney (2008) found that of 28 health-promoting media literacy interventions considered, only 10 
interventions incorporated all of the five core concepts. In general, effective interventions were 
slightly more likely than ineffective interventions to have taught all the core concepts (Bergsma & 
Camey, 2008). However, the differences in efficacy the authors find may also be due to the fact that 
effectiveness was defined differently in each of the studies included in the review. Based on a 
synthesis of the literature on media literacy education across contexts and applications, Martens 
(2010) note that while some media literacy programs focus on literacy-related outcomes such as 
critical and infonned consumption of media messages (see also Kubey, 2004), other programs 
“mostly try to increase children’s and adolescents’ mass media knowledge and skills because this, in 
turn, will maximize positive media effects and minimize negative ones” (Martens, 2010, p. 7). That is, 
many media literacy interventions are based on the assumption that individuals can protect themselves 
against negative influences from the media by becoming more media literate (Potter, 2004, 2010), and 
therefore the efficacy of such interventions ought to be assessed in relation to their ability to activate 
cognitive defenses against persuasive media content that promotes unhealthy behaviors. 

In reflecting on the challenge of evaluating the effect of health communication interventions, 
Hornik and Yanovitzky (2003) argue that a more robust approach for evaluating the efficacy of 
interventions is to focus on their underlying theory of change (or logic model). That is, the most 
credible form of evidence about the efficacy of interventions involves demonstrating that the program 
was able to cause a range of intended effects in audiences. Indeed, some of the most robust evidence 
regarding the effect of media literacy programs on adolescents’ health-related cognitions and behavior 
has been produced by Austin and colleagues using the message interpretation process (MIP) model 
(e.g., Austin, 2007; Austin & Knaus, 2000; Austin & Meili, 1994; Austin, Pinkleton, & Fujioka, 

2000), precisely because their assessments of direct program effects were tested against the prediction 
of their hypothesized model. In what follows, we discuss the contribution of the MIP model to the 
understanding of the process by which media literacy interventions influence their target audience and 
how it may be augmented, conceptually and empirically, to allow for more rigorous evaluations of 
effects. 


Effects of Media Literacy Interventions on Behavior: A Conceptual Framework 

To date, the field of media literacy has not yet produced a systematic account of the process by 
which media literacy interventions may contribute to positive health outcomes among adolescents 
(Martens, 2010). Representing this process necessarily requires an explication of how audiences 
interact (or are expected to interact) with the content and activities of media literacy programs. The 
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MIP model (Austin, 2007; Austin & Knaus, 2000; Austin & Meili, 1994; Austin et al., 2000), that has 
evolved from the integration of social cognitive theory (Bandura, 1986), expectancy theory of 
behavior (Fishbein & Ajzen, 1975), and dual-process theories of attitude change (Petty & Cacioppo, 
1986), attempts to do just that by tracking how adolescents process information they obtain through 
media literacy interventions to form their interpretation of media messages that promote unhealthy 
behaviors. According to this model, whether or not adolescents are influenced by media messages 
depends on the thoughts and feelings media messages invoke in them. In general, adolescents’ 
evaluation of media messages is informed by logic (comparison of the information provided against a 
benchmark) but also partially based on their affective reactions to the message. The logical 
interpretation of messages begins by evaluating the degree of realism or representativeness of media 
portrayals of the behavior and, unless it is rejected up front, is next evaluated for similarity (i.e., 
assessment of how closely the portrayal reflects nonnative personal experiences). Assuming that 
media portrayals are judged to match the adolescent’s perception of reference group nonns (i.e., high 
similarity), the next step in the process involves identification with the portrayal, which in turn, leads 
the adolescent to form positive expectancies (i.e., the expectation that doing something consistent with 
what is portrayed in the media message will have positive outcomes). The affective route of message 
interpretation that adolescents employ to evaluate the desirability or undesirability of the behavior 
portrayed in the media message is assisted by heuristics (e.g., liking of the peer model or the social 
scene portrayed in the ad) and, depending on the intensity of the affective reaction induced by the 
message, may bypass or bias the effect of logical evaluations on behavioral expectancies (Austin, 
Pinkleton, & Funabiki, 2007; Nathanson, 2004). Over time, as adolescents are exposed repeatedly to 
the same portrayal of behavior, they either internalize or reject it based on their interpretation. If they 
internalize the message, they perceive positive outcomes of the behavior promoted. If they reject it, 
they perceive negative outcomes of the behavior promoted. Thus, adolescents’ decisions to enact or to 
avoid the behavior portrayed in media messages can be reliably predicted from their perceived 
expectancies. 

Consistent with this theoretical rationale, research utilizing the MIP model generally 
found that media literacy interventions can influence adolescents’ decisions regarding risky health 
behaviors portrayed in the media (e.g., Austin et al., 2005; Chen, 2009; Kupersmidt, Scull, & Austin, 
2010; Pinkleton, Austin, Cohen, Miller, & Fitzgerald, 2007). However, because the model is primarily 
occupied with the interpretation of media messages, it does not fully represent the process by which 
participation in media literacy interventions may logically cause adolescents to reject or resist 
negative influences from media content. What is currently missing from the conceptualization of this 
process is an account of what aspect of the nature of the media literacy intervention may motivate 
adolescents to engage in the cognitive process of message interpretation as well as an account of when 
or under which circumstances adolescents’ critical evaluation of media portrayals will go on to 
influence their own decisions and behavior. Our discussion below highlights the crucial role of 
personal engagement with and reflection on the knowledge and skills adolescents acquire through 
their participation in a media literacy program regarding their ability to resist negative influences from 
the media. 


The Importance of Engagement and Reflectiveness 

As we consider the process by which media literacy interventions may influence adolescents’ 
decisions and behaviors in response to media portrayals, it is important to keep in mind that the target 
audience for these interventions are active participants in this process (Biocca, 1988). Audience 
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selectivity (i.e., selective attention, perception, and retention) and audiences’ degree of involvement 
with the content of media messages have long been recognized in persuasion theory to be important 
determinants of how people process and evaluate the infonnation to which they are exposed, and 
consequently, the likelihood that they will be influenced by this information (Petty & Cacioppo, 1986; 
Sherif & Hovland, 1961). For example, social judgment theory (Sherif & Hovland, 1961) explains 
that the likelihood of a message being accepted or rejected by audiences is a function of their ego- 
involvement with the issue, or the extent to which the information is judged to be personally relevant. 
Higher levels of ego-involvement make for a much narrower (or more highly selective) range of 
positions that a person finds acceptable. The elaboration likelihood model (ELM; Petty & Cacioppo, 
1986) defines involvement with the message as the amount of cognitive effort a person invests in 
evaluating messages, under the assumption that a greater investment of cognitive resources (or 
elaboration) is more conducive to learning from a message. Thus, audience involvement with 
messages is primarily assumed to be a function of an audience’s motivation to process the infonnation 
provided. We can therefore predict that if participants in a media literacy program judge the activity 
as boring or irrelevant, they will likely ignore or disregard the information it offers despite being 
exposed to it. On the other hand, if they find the activity interesting and relevant, they are more likely 
to process and remember the infonnation provided through their participation in this activity and, 
therefore, are more likely to use this information to form their attitudes and to guide their behavior. 

Given this, and because media literacy programs are essentially designed to be active and 
participatory (Hobbs, 2001, 2006; Masterman, 2001), it seems important to evaluate the degree to 
which target audiences’ are personally engaged with the activity to better predict the influence that 
this will have on their cognitions and behavior. Conceptually, our definition of engagement (see 
Figure 1) includes the dimensions of personal involvement with and perceived novelty of the activity, 
as both has been shown to be reliable predictors of information processing (Cacioppo, Petty, 

Feinstein, & Jarvis, 1996). Our conceptualization of personal involvement as the degree to which 
participants found the media literacy activity to be personally relevant and interesting without getting 
distracted is derived directly from its conceptualization in dual-process theories of persuasion (Petty, 
Cacioppo, & Goldman, 1981). Novelty is typically discussed in relation to the need for cognition 
component of dual-process models (see Cacioppo, Petty, Feinstein, & Jarvis, 1996). However, in the 
context of media literacy interventions it seems more reasonable to expect that perceived novelty will 
be assessed in relation to how activities are similar to or different from the typical educational 
activities that adolescents routinely experience. For example, Balch (1998) found that adolescent 
smokers would likely avoid anti-smoking programs that used approaches such as lecturing, preaching, 
nagging, and pushing that adolescents expected in these programs. Instead, Balch (1998) suggested 
that creative advertising opportunities may appeal to students and can involve students in designing 
such anti-smoking or quitting efforts/programs (see active involvement interventions, Greene, 2013). 
Conceptually, our definition of novelty measures the degree of newness or originality of the media 
literacy activity under the assumption that greater perceived novelty increases degree of adolescent 
engagement with the activity. 

Reflectiveness. Audiences’ degree of engagement with media literacy programs is a necessary 
condition for initiating the acquisition of knowledge and critical thinking skills that adolescents need 
to interpret media messages and portrayals but may be insufficient for influencing adolescents’ health- 
related decisions and behaviors unless the knowledge acquired is used for self-regulating one’s 
behavior. That is, whereas the MIP model views the process by which message interpretation 
influences behavioral expectancies as primarily involving conformity to reference group norms, social 
cognitive theory (Bandura, 1986), on which this model is based, also acknowledges the importance of 
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self-reflection to the potential for individuals self-regulating their behavior based on assessment of 
information they receive from their environment. According to Bandura (1986), people are self- 
motivated to change their behavior once they accept that their current behavior or conduct does not 
meet their personal standards or goals. When they perceive a discrepancy between their current 
behavior and ideal behavior (which is often accompanied by feelings of discomfort or affective 
reaction), they are motivated to reflect on how to align their personal behavior or goals with the 
standards they set for themselves. Assuming they have a sense of self-efficacy (that is, a belief in their 
ability to change their behavior), they make specific plans for changing their behavior that they then 
go on to execute. 

Based on the self-regulation rationale of social cognitive theory, we propose that the degree to 
which participants personally reflect on what they learned from the media literacy intervention will be 
an important determinant of whether or not this knowledge is processed in a manner that can influence 
their behavioral expectancies and, subsequently, their behavior. Accordingly, we conceptualize 
reflectiveness as composed of two dimensions (see Figure 1): critical thinking (the degree to which 
the media literacy activity stimulated critical evaluations of media messages) and personal reflection 
(the degree to which the knowledge acquired was used to reevaluate personal conduct). 


Figure 1 

Conceptual Model of Engagement and Reflectiveness 




Study Goals. The primary goal of this study was to develop and test theory-grounded and 
reliable measures of audience engagement with and reflectiveness on media literacy interventions to 
enable the incorporation of these constructs into evaluations of media literacy interventions as 
potentially important explanatory variables. We examine and demonstrate the predictive utility of 
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including both constructs into the outcome evaluation of a media literacy intervention elsewhere 
(Greene et ah, 2011). Our focus here is on describing the procedure we followed to develop and test 
the measurement model presented in Figure 1. 

Consistent with our conceptualization of each construct, we sought to represent engagement 
by measuring two distinct dimensions of this construct: degree of involvement with the intervention 
and degree of perceived novelty of the intervention. We therefore sought to develop two 
corresponding sub-scales of self-reported items that are moderately correlated with one another 
(because this finding will be consistent with the argument that each is a distinct dimension of the same 
latent construct). Similarly, we sought to represent reflectiveness via the combination of two 
dimensions (or subscales) - critical thinking and personal reflection - that are moderately correlated. 
Finally, because we expect reflectiveness to be dependent in part on degree of engagement with the 
intervention based on the logic of the self-regulation in social cognitive theory (Bandura, 1986), the 
model in Figure 1 also includes a hypothesized link (or an expectation of association) between the two 
constructs. In the next section we describe the measurement of each of these constructs and the 
methodology we used to evaluate the validity, reliability and overall utility of the instrument based on 
data obtained from an actual media literacy intervention. 

Research Methods 

The project utilizes data collected from two samples of adolescents to evaluate responses to a 
media literacy curriculum. We describe the combined procedure first, followed by the two samples, 
the measures, and finally data analyses. This measurement study is part of a larger project testing an 
intervention in which subjects were randomly assigned to different conditions. However, for the 
purposes of this paper, data were combined across all conditions to ensure adequate control for type I 
and type II errors, after we verified that the measurement structure was invariant across conditions. 

Procedure. Participants in both samples attended a 75-minute media literacy workshop that 
included discussion of persuasion techniques and advertising claims employed by advertisers, along 
with coverage of the production features typically used to sell alcohol and other products. Participants 
learned to critically evaluate the claims made in alcohol ads and in anti-alcohol ads. At the end of the 
workshop, participants completed a short questionnaire measuring their evaluation of the media 
literacy intervention, in addition to offering oral comments (15 additional minutes taken for the 
evaluation). This paper focuses on the curriculum evaluation measures of the immediate post¬ 
implementation questionnaire. 

Sample I. The data used for Sample I of this paper were collected in 2010 as part of a posttest- 
only pilot study to adapt and refine an existing brief media literacy intervention targeting adolescents. 
The full sample included 308 students from 32 schools across Pennsylvania attending a Leadership 
Institute (representing rural, smaller town, smaller city, suburban, and urban school districts). After 
removing 14 participants due to missing data on the relevant items, the final sample included 294 
male (n = 44) and female (n = 169; 49 students did not report gender) 10 th grade high school students 
(ages 14-16; M= 15.62, SD = .57). About 64% reported their race/ethnicity as White, with others 
indicating Hispanic/Latino (17%), African American (13%), American Indian/Alaskan Native (3%), 
Asian American/Pacific Islander (3%), or some other race/ethnicity (less than 1%). 

Sample II. The data used for Sample II came from a longitudinal feasibility test of the revised 
curriculum implemented in 2011. After removing three participants due to missing data on the 
relevant items, the final sample included 171 male (n = 59) and female (n = 102; 10 students did not 
report gender) 10 th grade high school students (ages 14-17; M= 15.75, SD = .90) from 34 schools 
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across Pennsylvania (representing rural, smaller town, smaller city, suburban, and urban school 
districts) who participated in this study while attending a Leadership Institute. About 70% reported 
their race/ethnicity as White, while others indicated Hispanic/Latino (10%), Asian American/Pacific 
Islander (10%), American Indian/Alaskan Native (4%), African American (3%), or some other 
race/ethnicity (3%). 


Measures 

The present analyses focus on 14 questionnaire items (expanded to 16 in Study II) that 
measured participants’ degree of engagement with the media literacy workshop and their subsequent 
reflection on what they learned. Tables 1 and 2 list the items that were developed to measure 
engagement (see the subscales labeled “involvement” and “perceived novelty”) and reflectiveness 
(see the subscales labeled “critical thinking” and “personal reflection”). The items were initially 
developed during the pilot implementation of the intervention and underwent refinement based on the 
feedback collected from participants (see Greene et al., 2011). Most items were five-point Likert type. 
Scales were modified slightly between Study I and Study II. Specifically, the items measuring critical 
thinking were modified to include a five-point scale for consistency with the other subscales in the 
curriculum evaluation measure, and two additional items were developed for perceived novelty and 
reflectiveness. Since two Study I items did not correlate well with the scale, they were modified for 
Study II. 

Data Analysis: Study I. In the first step of the analyses, we examined the distribution of all 
items to detect any irregular patterns of responses. All items exhibited approximately nonnal 
distribution of responses (as evidenced by kurtosis and skewness values significantly smaller than 2). 

We then examined inter-class correlations within subscale items for each of the four subscales. In the 
next step, we subjected the data to principal component analysis, using eigenvalues, scree plots and 
conceptual framework to detennine the number of factors and then utilized Varimax rotation to 
extract the factor structure and create subscales. Each of the subscales’ reliability was assessed using 
Cronbach’s alpha as a measure of internal consistency of the set of items making up each one of the 
subscales (an adequate alpha level was .70, good .80). Finally, we examined the correlations among 
the four subscales to assess the convergent and discriminant validity of our measure. 

Data Analyses: Study II. The measures were modified slightly for Study II including generation 
of two additional items (one for perceived novelty and one for reflectiveness). In the first step of the 
analyses for Study II, we examined the distribution of all items to detect any irregular patterns of 
responses. All items exhibited approximately normal distribution of responses. Inter-class correlations 
among items within a subscale were examined. Because sub-scales were explored and identified based 
on the data collected from the first sample, our goal was to confirm the factor structure of all variables as 
subscales of the latent variable of motivation to process information using second-order confirmatory 
factor analysis (CFA) using maximum likelihood structural equation modeling (AMOS 18.0). The 
reliability of each of the subscales was assessed using Cronbach’s alpha. See Note 1 below. Finally, we 
examined the correlations across the four different subscales to assess the convergent and discriminant 
validity of our measure. In the next section, we present the results from Study I and Study II. 

Results 

Study I. Table 1 summarizes the findings of exposing the data to principal component 
analysis. This procedure identified four factors from the 14 items based on examination of the scree plot 
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and the best conceptual fit (see Varimax-rotated factor loadings of individual items in Table 1). The first 
factor, labeled Involvement (eigenvalue = 5.26), was composed of 4 items and accounted for 35.6% of 
the variance with good internal consistency (a = .85). The second factor, labeled Personal Reflection 
(eigenvalue = 2.05), was composed of 4 items and accounted for 14.6% of the variance with acceptable 
internal consistency (a = .75). The third, labeled Perceived Novelty (eigenvalue = 1.16), was composed 
of 3 items and accounted for 8.4% of the variance with acceptable reliability (a = .75). The last (fourth) 
factor extracted from the data, labeled Critical Thinking (eigenvalue = 1.04), was also composed of 3 
items and accounted for 7.4% of the variance with acceptable reliability (a = .76). This analysis 
confirmed our underlying theoretical rationale that that the motivation to process the infonnation 
provided by the intervention is a multidimensional construct. 

Table 1 

Study I: Workshop Engagement and Reflectiveness Reliability, Item Means and Primary Factor Loading 


Scale/Item 

M 

SD 

Factor Loading 

Involvement (1 = Strongly Disagree to 5 = Strongly Agree) a = .85 

-The workshop was interesting to me. 

3.78 

.80 

.79 

-I got easily distracted during the workshop. (R) 

3.44 

.97 

.78 

-I enjoyed this workshop. 

3.46 

.85 

.79 

-The workshop was boring. (R) 

3.61 

.98 

.79 

Personal Reflection (1 = Not at all to 4 = A Lot ) a = .75 

-How much did the workshop make you think about the impact of advertising 

3.00 

.80 

.61 

on you personally? 

-How much did the workshop make you think about the impact of advertising 

3.08 

.81 

.67 

on your peers? 

-How much did the workshop make you think about your alcohol use? 

2.54 

1.11 

.74 

-How much did the workshop make you think about how your peers use 

2.98 

.94 

.75 

alcohol? 

Perceived Novelty (1 = Strongly Disagree to 5 = Strongly Agree) a = .75 

-The workshop was just like what we normally do in school. (R) 

3.45 

1.06 

.78 

-I’ve never done anything like what we did in the workshop today. 

2.43 

1.06 

.72 

-The workshop was different from regular school classes. 

3.53 

1.05 

.85 

Critical Thinking (1 = Strongly Disagree to 5 = Strongly Agree) a = .76 

-This workshop said something important to me. 

3.46 

.89 

.61 

-The messages in the workshop made me think about the ads that I see. 

3.79 

.78 

.83 

-The messages in the workshop made me think about the truthfulness of ad 

3.80 

.77 

.86 


claims. 


Once a composite variable was created to represent each subscale by averaging responses to 
relevant items, the next step of the analysis involved an examination of the bivariate correlations 
between the four factors (or subscales) to establish convergent validity. Convergent validity is 
established when measures of constructs that theoretically should be related to each other (such as 
subscales of the same latent variable) are, in fact, correlated with each other, albeit not perfectly (or else 
they represent the same dimension of the construct). As Table 3 illustrates, the non-trivial and 
statistically bivariate correlation coefficients among the different subscales are consistent with our 
expectation regarding the convergent validity of our instrument. Specifically, the correlations between 
the two subscales measuring the same latent construct - involvement and perceived novelty (r = .48) 
that measure engagement, and critical thinking and personal reflection (r = .56) that measure 
reflectiveness - were generally greater than the correlations between pairs of subscales hypothesized to 
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measure different constructs. The discriminant validity of our instrument is established by the small 
correlations observed between the subscales of involvement and those that measure personal reflection. 
At the same time, the fact that statistically significant moderate correlations exist between subscales of 
engagement and reflectiveness (r = .39), suggest that both variables are positively associated as 
predicted by the self-regulation rationale of social cognitive theory. 


Table 3 

Bivariate Correlations, Study 1 



1 

2 

3 4 

1. Involvement 

— 



2. Perceived Novelty 

.48** 

-- 


3. Personal Reflection 

.41** 

.18** 

... 

4. Critical Thinking 

.47** 

.24** 

.56** 


*p < .05 (two-tailed), ** p < .01 (two-tailed) 


Study II. Table 2 summarizes the results of testing the conceptual measurement model in Figure 1 with 
a second independent sample using a confirmatory factor analysis procedure. Specifically, a second-order 
CFA procedure was employed to confirm the four-factor structure that was extracted from the principal 
component analysis of Sample I with a second independent sample. The model fit statistics for the second- 
order factor analysis on all four scales confirmed good fit for the four-factor structure [y 2 (41) = 86.22, %/df= 
2.10 ,p> .001; CFI = .90, RMSEA = .08] as subscales of the same latent variable (motivation to process the 
information provided in the media literacy workshop). The estimated factor loading of each item is reported in 
Table 2. 

A comparison of the scale reliability estimates (alphas) reported in Table 2 to the comparable 
reliability estimates in Table 1 suggests that the slightly refined set of survey items used in Sample II 
improved the overall reliability of each subscale, with the exception of the critical thinking subscale (where 
alpha = .75 in Sample I but .69 in Sample II). However, inspecting the patterns of means and standard 
deviations on these items across the two samples, we note that there was a greater homogeneity in responses 
to this subscale among Sample I participants in comparison to Sample II participants, so the diverging 
estimates of reliability are likely due to between-samples variations, particularly when the wording of the 
items has not changed between the two applications of the instrument. Overall, then, the survey items used 
make for reliable measures of the subscales of interest. 

Finally, as we did with the data in Sample I, we estimated a bivariate correlation matrix between the 
composite variables measuring the four subscales to estimate the convergent and discriminant validity of the 
four subscales (see Table 4). As expected theoretically, and consistent with the pattern observed for Sample I 
(see Table 3), correlations among the four subscales were positive and statistically significant, with 
correlations between each pair of subscales that measure the same latent construct being stronger than 
correlations of subscales that measure different latent constructs (i.e., engagement and reflectiveness). 
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Table 2 

Study II: Workshop Engagement and Reflectiveness Reliability, Item Means and Primary Factor- 
Loading 


Scale/Item 

M 

SD 

Factor Loading 

Involvement (1 = Strongly Disagree to 5 = Strongly Agree ) a = .87 




-The workshop was interesting to me. 

3.50 

.90 

.88 

-I got easily distracted during the workshop. (R) 

3.01 

1.09 

.88 

-I enjoyed this workshop. 

3.51 

.88 

.79 

-The workshop was boring. (R) 

2.76 

1.12 

.54 

-This workshop material was important to me. 

3.50 

.86 

.63 

Personal reflection (1 = Strongly Disagree to 5 = Strongly Agree) a = .69 




-The workshop made me think a lot about the impact of advertising 

3.58 

.80 

.94 

on me. 




-The workshop made me think a lot about the impact of advertising on my 

3.64 

.82 

.59 

peers. 




-The workshop made me think a lot about my alcohol use. 

3.02 

1.23 

.46 

Perceived Novelty (1 = Strongly Disagree to 5 = Strongly Agree) a = .77 




-The workshop was just like what we normally do in school. (R) 

2.76 

1.15 

.63 

-I’ve never done anything like what we did in the workshop today. 

2.63 

1.13 

.63 

-The workshop was different from regular school classes. 

3.40 

1.08 

.82 

-The workshop was unique. 

3.37 

.96 

.66 

Critical Thinking (1 = Strongly Disagree to 5 = Strongly Agree) a = .83 




-The workshop made me think about the ads that I see. 

3.87 

.79 

.87 

-The workshop made me think about the truthfulness of ad claims. 

3.83 

.83 

.78 

-The workshop made me think about advertising. 

3.80 

.78 

.72 


Thus, involvement and perceived novelty correlated strongly (r = .43) as did critical thinking and 
personal reflection (r = .85), suggesting convergent validity of our measurement instrument. In contrast, 
correlations between subscales that belong in different constructs were much smaller. As before, we interpret 
the significant correlation between involvement and personal reflection (r = .37) to suggest that 
reflectiveness depends to a considerable degree on a person’s engagement with the media literacy workshop. 

Table 4 

Bivariate Correlations, Study 2 

_1_2_3_4_ 

1. Involvement 

2. Perceived Novelty .43** — 

3. Personal Reflection .37** .16* — 

4. Critical Thinking .45** .24** .85** 


*p < .05 (two-tailed), ** p < .01 (two-tailed) 
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Discussion 

We propose that a more complete understanding of the cognitive process through which participation 
in media literacy interventions causes the intended outcomes in participants is necessary to harness the full 
potential of these programs. Based on the self-regulation rationale of social cognitive theory (Bandura, 1986) 
and dual-process models of infonnation processing (Petty & Cacioppo, 1986), we highlight the potential 
importance of the degree to which audiences are cognitively engaged with and reflect on the knowledge they 
obtain through these programs to the explanation and prediction of the effects that these programs have (or 
fail to have) on adolescents’ decisions and behaviors. However, because theory-grounded and reliable 
measures of these constructs in the context of media literacy programs are not presently available, the primary 
goal of this study was to develop and test valid and reliable survey measures of engagement and 
reflectiveness. The measurement model we developed (see Figure 1) recognizes the multi-dimensionality of 
these constructs, with engagement encompassing involvement with the activity and its perceived novelty, and 
reflectiveness being represented by critical thinking about the topic of the activity as well as personal 
reflection on the insights generated from participation. 

Overall, the results of two independent tests of this particular measurement model in the context of 
evaluating a media literacy intervention lend considerable support to this model. As expected, the factor 
analytic procedure confirmed that the proposed subscales provide adequate representation of the constructs of 
interest (as evidenced by the convergent and discriminant validity of the instrument) as well as the hypothesis 
that reflectiveness is dependent on engagement to some degree (by virtue of engagement preceding personal 
reflection). Despite being tested with a specific group of adolescents that may not be representative - 
demographically, developmental^, or in tenns of motivation of the broader and diverse group of adolescents 
who may be susceptible to influences from media messages and portrayals, we believe that instrument we 
developed can be used reliably to measure the same constructs within this broader group and in the specific 
context of media literacy interventions. However, it is also clear that the instrument can benefit from further 
refinement and testing, as well as the inclusion of factors, such as self-efficacy, that social cognitive theory 
posits as important moderators of the degree to which acquired knowledge and skills translate into decisions 
and behavior via the process of self-regulation. 

We believe that this work has both theoretical and practical implications. Theoretically, it signals the 
importance of expending the conceptual framework used to assess the effect of media literacy interventions on 
their audience by tracking a process of change in personal predispositions, decisions, and behaviors in 
response to influence from media messages that involves self-regulation as envisioned by social cognitive 
theory, and which augments the nonnative or social identification-based process of influence that the MIP 
model (Austin, 2007) proposes and that existing empirical tests support (e.g., Austin et al., 2005; Banerjee & 
Greene, 2006, 2007; Pinkleton et al., 2007). We believe that incorporating both processes into the evaluation 
of media literacy interventions can produce a more complete understanding of how and under which 
circumstances these interventions can be a viable tool of influencing decisions and behaviors among 
adolescents as well as target audiences more generally. Practically, this work provides researchers with a valid 
and reliable instrument for measuring audiences’ degree of engagement with media literacy interventions (as a 
prerequisite for expecting intervention effects) and personal reflection on the experience (which is necessary 
for self-regulation), that can be easily adapted to the particular context of the intervention, and be used (in 
addition to existing measures that track the influence of the program) for formative, process, and outcomes 
evaluation. We hope this this work will encourage media literacy scholars to further engage with the question 
of causal paths and to develop additional measures that will advance rigorous evaluations of media literacy 
interventions that are theoretically-driven. 
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